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Abstract Text (1) : 

A robotic apparatus and method of acquiring jumbled objects or workpieces from a 
bin, and transferring them to a final site, or to an intermediate site for quality 
inspection. A video camera system produces a video signal representing a brightness 
image of the jumbled objects. An intermediate amplitude range of the video signal 
is selected, expanded, stored, and quantized into digital pixels. A computer 
enhances the image using histogram techniques, performs edge suppression, performs 
repeated shrinking with progressively-increased pixel thresholds until an area 
limit is reached, performs clustering of closely-spaced pixels, and commands 
movement of the robot gripper to a computed grip site on a recognized object. The 
gripper may limit its gripping pressure as a function of the deformation of the 
object gripped. 

Application Filing Date (1) : 
19840228 

Drawing Description Text ( 9) : 

FIG. 6 is a flowchart describing the operation of the system during the automatic 
training phase ; 

Drawing Description Text (10) : 

FIG. 7A is a flowchart describing the overall operational phase including 
communication with the robot; 

Detailed Description Text (8) : 

In order to solve the above problems, an intensity histogram can be made by 
analyzing the scene within a window. The bi-modal distribution of an intensity 
histogram depicts the nature of scene illumination and the reflection 
characteristics of workpieces. For bright objects on a dark background, two clear 
peaks can be obtained. One peak corresponds to shadows cast by jumbled workpieces 
on one another and the background, while the other peak corresponds to reflection 
from the brightly illuminated workpieces. In this case, the initial threshold 
should be selected at a point between the two peaks but closer to the brighter 
peak, as judged during the training phase . Such a judicious choice of starting 
threshold can speed up the process significantly. 

Detailed Description Text (11) : 

As shown in FIG. 4A, histogram H.sub.a 64 incorporates an ordinate y==number of 
pixels of a given brightness, and the abscissa x=gray scale of light intensity, 
generally shown in binary form with two major peaks. The left one represent shadows 
and the right one strong reflections of the incident light as well as some ripples 
due to random reflections from the jumbled objects. When this histogram data is 
filtered to minimize the ripples so as to more clearly identify the desired two 
peaks, histogram H*a at 65 results. The latter is then processed to determine the y 
(x) amplitude which has y(x+l) and y(x-l) both less in amplitude than y(x), thus 
identifying y(x) as a peak. It is then shown as a single line in H"a at 66. In a 
similar fashion, FIG. 4B shows a histogram Hb of an illuminated object which has a 
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variety of reflection surfaces and black or gray background, therefore having many 
peaks. The signal appears as H.sub.b * at 68 after filtering, as H.sub.b " at 69 
after peak detection, and as H.sub.b at 70, after an ordering of the magnitude 
of the various peaks, the two largest peaks y.sub.3 and y.sub.5 being selected. The 
threshold T.sub.o is usually adjusted between x.sub.3 and x.sub.5 to optimize the 
selectivity between individual objects. In H.sub.a it is seen that y.sub.l is the 
dark peak and y.sub.2 is the bright peak. The threshold T.sub.o is derived by 
experiment during the training phase as follows: 

Detailed Description Text (16) : 

K may vary from 0.3 to 1.0 depending on the light situation, the surface quality of 
the object and the optimum performance obtained during the training phase but under 
unusual conditions may have a greater range (e.g., to 1.25). Equipment actually 
constructed used a value of K equal to 0.9. 

Detailed Description Text (18) : 

When the digital video data within the selected window output from 192 is detected 
in a 256. times. 16 RAM data buffer 193 its binary value is used as an absolute 
address to the memory and the resulting location within the RAM is incremented by 
one. This process continues over the entire windowed field of view. When completed, 
each location of the RAM represents one of 256 possible intensity values and the 
data within this location represents the number of pixels of this intensity which 
have occurred. The RAM can then be accessed through the multibus interface 194 by 
the host computer 28 for analysis of the histogram data. The width of the histogram 
memory is 16 bits to accommodate the worst case condition where a 256 . times . 256 
pixel window contains all the same intensity values (65,536 in the exemplary 
equipment) . 

Detailed Description Text (34) : 

Intensity difference at the edges of touching objects. 
Detailed Description Text (42) : 

At 101, the program, having used this data, determines the window size and sequence 
and sends the size and shape information to the robot controller 102. In 103, the 
camera 1 feeds a picture of the surface of the bin containing the objects. The 
vision controller processes it in the video range expander 14, stores it in the 
frame grabber 11, establishes a window and derives an intensity histogram of a 
subimage inside this window. From this is derived an adaptive threshold in 104, 
which is used to enhance the image of the object or objects against the background. 
The program then computes the area, A.sub.o, the number of pixels in the bright 
portions of the object under scrutiny. A decision is then made in 105, as to 
whether the objects are separated, or jumbled and in contact with each other, based 
on operator input data. In the former, the no decision of 105 leads into a decision 
at 111 as to whether there remain other windows to be processed. If so, this cycle 
is repeated until all windows have been done in a similar fashion. If the last 
window has been processed the computer in 112 determines the mean area .mu..sub.A 
of the bright portions in all the windows as well as the standard deviation, 
.delta. .sub. A, of the areas, and computation is made of . eta . = . mu . . sub . A - 
Q.sub.A . Sigma. . sub. A, which is the lower limit of tolerable area to be allowed 
during the operational mode. Q.sub.A is a scale factor. These data are stored in 
the computer memory. In 113, the x, y coordinates and the orientation of the 
selected objects are displayed on the monitors and the training data is checked for 
accuracy by the use of the operational mode of FIGS. 7A, 7B, and 7C. If the 
displayed object location and orientation data are inaccurate, successful 
acquisition of an object by the robot is not possible. The training cycle must then 
be repeated until successful acquisition can be attained and the appropriate data 
is stored in the computer for use during system operation to be described in 
connection with FIG. 7. 

Detailed Description Text (47) : 
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The standard deviation is ##EQU2## is also calculated and stored in memory as well. 



Detailed Description Text (51) : 

FIG. 7A is a flowchart of the operational phase of the equipment, showing in detail 
what is included in box 97 of FIG. 5, the total system diagram. The operational 
phase uses constants derived during the training phase of FIG. 6. FIGS. 7B and 7C 
show in greater detail what is included in box 117 of FIG. 7A. 

Detailed Description Text (54) : 

The flowchart of FIG. 7B starts at 130 with the taking of a picture of the objects 
on the surface of the bin, and the feeding of the video signal to 131, where the 
video range expander, whose circuit is shown in FIG. 3A, processes it by using the 
stored information on black level and offset setting information from 132 generated 
during the training phase of FIG. 6. This processed video is stored at 133 in the 
frame grabber. To ensure that the process is proceeding properly, the resulting 
gray scale image is displayed at 134 on the monitor. 

Detailed Description Text (84) : 

where I(x,y) and I (x. sub. 2 y) are the reflected light intensities from adjacent 
pixels along a scan line. The above intensity difference has a basic limitation in 
that the reflected intensity is dependent on the incident light, I.sub.L, as 
follows : 

Detailed Description Text (93) : 

which has been calculated during the training phase described in FIG. 6, and is fed 
from 147 into 146. Or, (2) the number of iterations m=N.sub.L, which is a fixed 
limit such as 10 in the exemplary system. These limits prevent wasting computer 
time during object acquisition. Because the image at iteration m has too few usable 
bright areas, the data at the previous iteration, m-1, is selected and used at 148. 

Detailed Description Text (95): 

There are, however, two further criteria which must be met by bright areas before 
being designated as candidates for acquisition by the robot. As shown in 153, the 
clusters must possess areas greater than a minimum value, S.sub.min and elongations 
(E=Length/Width) greater than E.sub.min, which have been established during the 
training phase 154 for elongated objects. Those clusters meeting these criteria are 
then identified to 120 in FIG. 7A for further computation. 

CLAIMS : 

1. A robot system for acquiring randomly-arranged workpieces, comprising 

a robot assembly including a hand, and a controller for controlling the movement of 
the hand to positions to successively grasp workpieces of given size and transfer 
them to a predetermined site, 

a video imaging device to provide a video signal representing a brightness image of 
the randomly-arranged workpieces, and 

a computer programmed to be responsive to the video signal from the imaging device 
and operative to modify the image, to compute a holdsite on a workpiece in the 
image, and to command the robot assembly to grasp the workpiece and transfer it to 
said predetermined site, 

said video imaging device including a video range expander for selecting and 
amplifying a range of video signal amplitudes between a gray threshold level and a 
white clipping level. 
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said computer including program means to quantize the selected video signal having 
said range of amplitudes into digital pixels within a plurality of windows all 
having a size proportional to the size of the workpieces, said means to quantize 
the selected video signal including means to generate the same number of digital 
pixels of the image within the window regardless of the size of the window, and to 
enhance the quantized image by making all pixels having a brightness intensity less 
than a threshold intensity T equal to zero, and retaining the brightness 
intensities of all pixels above the threshold T, said threshold intensity T being 
computed by program from an intensity histogram so that T exceeds the intensity of 
the most numerous dark pixels by a predetermined fraction of the difference between 
the intensity of the most numerous dark pixels and the intensity of the most 
numerous bright pixels. 

3. A system according to claim 2 wherein the brightness ratio of adjacent pixels is 
computed by determining the difference of the logarithms of the reflected 
intensities of the two pixels. 

7 . A system according to claim 6 wherein said predetermined ratio of areas is 
computed during a training phase to be in the range from 1/2 the standard 
deviation, to three times the standard deviation greater than the average ratio of 
areas found in all of said plurality of windows by progressively shrinking the 
images in the plurality of windows until the ratio of each initial area to each 
final area is greater than a predetermined training ratio. 

14. A system according to claim 13 wherein said computer includes program means to 
compute the center of area of the cluster of pixels to determine the location of 
the grasp site, and to compute a minimum dimension of the cluster of pixels to 
determine the orientation of the grasp site. 

17. A robot system for acquiring non-touching randomly-arranged workpieces, 
comprising 

a robot assembly including a hand, and a controller for controlling the movement of 
the hand to positions to successively grasp workpieces of given size and transfer 
them to a predetermined site, 

a video imaging device to provide a video signal representing a brightness image of 
the randomly-arranged workpieces, and 

a computer programmed to be responsive to said video signal from the imaging device 
and operative to modify the image, to compute a holdsite on a workpiece in the 
image, and to command the robot assembly to grasp the workpiece and transfer it to 
said predetermined site, 

said video imaging device including a video range expander for selecting and 
amplifying a range of video signal amplitudes between a gray threshold level and a 
white coupling level, 

said computer including program means to quantize the selected video signal having 
said range of amplitudes into digital pixels within a plurality of windows all 
having a size proportioned to the size of the workpieces, 

program means to enhance the quantized image by making all pixels having a 
brightness intensity less than a threshold T equal to zero, and retaining the 
brightness intensities of all pixels above the threshold T, wherein said threshold 
intensity T is computed from an intensity histogram by program means which selects 
the threshold intensity T at a value intermediate that of the most numerous dark 
pixels and that of the most numerous bright pixels, and 

program means to determine whether the bright pixel area in a given window is 
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greater than an area, computed during a training phase, which is in the range of 
1/2 a standard deviation, to three standard deviations less than, the average of 
the areas found in all of said plurality of windows. 

22. A system according to claim 21 wherein said computer includes program means to 
compute the center of area of the cluster of pixels to determine the location of 
the grasp site, and to compute the axis of least moment of inertia of the cluster 
of pixels to determine the orientation of the grasp site. 

25. A robot system for acquiring randomly-arranged workpieces of given size, 
comprising 

a robot assembly including a hand, and a controller for controlling the movement of 
the hand to positions to successively grasp workpieces and transfer them to a 
predetermined site, 

a video imaging device to provide a video signal representing a brightness image of 
the randomly-arranged workpieces, and 

a computer programmed to be responsive to the video signal from the imaging device 
and operative to modify the image, to compute a holdsite on a workpiece in the 
image, and to command the robot assembly to grasp the workpiece and transfer it to 
said predetermined site, 

said video imaging device including a video range expander for selecting and 
amplifying a range of video signal amplitudes between a gray threshold level and a 
white clipping level, 

said computer including program means to quantize the selected video signal having 
said range of amplitudes into digital pixels within a plurality of windows all 
having a size proportioned to the size of the workpieces, 

program means to enhance the quantized image by making all pixels having a 
brightness intensity less than a threshold T equal to zero, and retaining the 
brightness intensities of all pixels above the threshold T, 

program means to form clusters of bright pixels by adding bright pixels in spaces 
less than a predetermined minimum distance between existing bright pixels, 

program means to identify said clusters having an area greater than a predetermined 
minimum area, 

program means to identify said large clusters which also have a length-width ratio 
greater than a predetermined minimum, 

program means to compute the location and orientation of a grasp site on a 
workpiece represented by an identified large elongated cluster of pixels, and to 
command the robot assembly to grasp the workpiece and transfer it to a 
predetermined site, and 

program means to compute the center of area of the cluster of pixels to determine 
the location of the grasp site, and to compute the axis of least moment of inertia 
of the cluster of pixels to determine the orientation of the grasp site. 

26. A system according to claim 25 wherein the computation of the center of area of 
pixels includes computing an initial center of area by finding the location of the 
pixel which occurs half way on a rasterscan through the cluster of bright pixels, 
and computing a corrected center of area by averaging the bright pixels on a small 
plurality of horizontal and vertical scans passing in the immediate vicinity of 
said initial center of area. 
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